Introduction to Machine Learning

Chapter 05: Classification Metrics and Threshold Evaluation

1. Introduction

Once we have a disciplined evaluation protocol, we still need to decide what performance means. A single number such as accuracy can be useful, but it can also be deeply misleading when the classes are imbalanced or when different types of errors have different costs.

This chapter develops the classification metrics needed to look inside a model's decisions: the confusion matrix, accuracy, precision, recall and the Fβ family. We then move from hard class labels to continuous scores, showing how changing the decision threshold trades off true-positive and false-positive rates and leads naturally to the ROC curve, AUC, model comparison, and operating-threshold selection.

Learning Objectives

2. Theory

Classification evaluation starts with the confusion matrix and the limitations of accuracy, then moves to precision, recall and Fβ before treating the classifier output as a continuous score and studying threshold trade-offs with ROC and AUC.

2.1 The Confusion Matrix and Accuracy

Every evaluation in this chapter is built on the standard classification accuracy, which is read off the confusion matrix:

Predicted Label
Positive (+)Negative (−)
True
Label		+True Positive (TP)False Negative (FN)
−False Positive (FP)True Negative (TN)
\( \text{Accuracy} = \dfrac{TP + TN}{TP + TN + FP + FN} \)

2.2 Class Imbalance — Why Accuracy Fails

All the evaluation so far has assumed that accuracy is a reasonable summary of performance. When one class is much rarer than the other, that assumption breaks down.

🚨 The "99% Accuracy" Fraud Detector Trap

Dataset: 9,990 legitimate transactions (= Class 0), 10 fraud (= Class 1). Total n = 10,000.

A trivial model that predicts "Legitimate" for every single transaction achieves 9,990 / 10,000 = 99.9% accuracy — and 0 frauds caught. High accuracy, completely useless.

PredictedTotal
0 (Legit)1 (Fraud)
True09,990 (TN)0 (FP)9,990
1 (Fraud)10 (FN)0 (TP)10
Total9,990010,000

Accuracy is always reported, but never trusted alone on imbalanced tasks.

2.3 Confusion Matrix Terminology (Medical framing is memorable)

To describe performance on an imbalanced problem we need to name the four cells of the confusion matrix separately. A medical screening test is a convenient way to remember them.

2.4 Precision, Recall, and the Fβ Family

From these four counts we can define metrics that, unlike accuracy, do not become meaningless when one class dominates the dataset:

\( \text{Recall (Sensitivity, TPR)} = \frac{TP}{TP + FN} \;\;=\;\; \frac{\text{caught positives}}{\text{all real positives}} \)

\( \text{Precision (PPV)} = \frac{TP}{TP + FP} \;\;=\;\; \frac{\text{caught positives}}{\text{all predicted positive}} \)

\( F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} \quad \text{(harmonic mean)} \)

\( F_\beta = (1 + \beta^2) \cdot \frac{\text{Precision} \cdot \text{Recall}}{\beta^2 \cdot \text{Precision} + \text{Recall}} \)

2.5 Why Harmonic Mean, not Arithmetic?

Intuitive example: Precision = 100%, Recall = 50%

Arithmetic mean = (1.0 + 0.5)/2 = 0.75, which looks acceptable. But a model that flags only one very obvious positive (so it makes no false positives and reaches 100% precision) while missing half of all real positives is not a 75% model. The arithmetic mean hides the poor recall.

Harmonic mean = 2 × 1.0 × 0.5 / (1.0 + 0.5) = 1.0/1.5 = 0.667. It is always ≤ the arithmetic mean, and it is dragged toward the minimum of the two — exactly what we want so we can't game the metric by doing great on one and terrible on the other.

Numeric check: (Precision=0.01, Recall=0.99). The arithmetic mean is 0.50, which again looks acceptable. The harmonic mean is ≈ 0.02, which is the correct verdict: the model is flagging almost everything as positive and is right only 1% of the time when it does.

The pattern holds for any two positive numbers, not just Precision/Recall. Take a = 1 and b = 9: the arithmetic mean is (1 + 9)/2 = 5, but the harmonic mean is 2·1·9/(1 + 9) = 18/10 = 1.8 — pulled sharply toward the smaller number. This is the same pull that makes F1 punish a model that is excellent on one of Precision/Recall but weak on the other.

2.6 From Class Labels to Probabilities

Most classifiers can return a continuous score, not just a binary label. For KNN with K=19, if 13 neighbors say + and 6 say −, then:

\( P(+ \mid \mathbf{x}) = \frac{13}{19} \approx 0.68 \)

The default threshold is to classify as + if P ≥ 0.5. There is nothing special about 0.5, however. The threshold should be chosen from the cost of each type of error in the application, not taken from convention.

2.7 Thresholds Trade Off TPR vs. FPR

How threshold changes affect TPR and FPR A vertical diagram showing that moving the classification threshold from zero to one decreases both true positive rate and false positive rate, with intermediate thresholds between them. Threshold moves from 0 → 1 A stricter threshold changes what the classifier calls “positive” 0.1 LOW THRESHOLD Predict + freely A small score is enough to trigger a positive prediction. High TPR catch most real +’s High FPR lots of false alarms increase threshold INTERMEDIATE THRESHOLDS ⋯ 0.2 · 0.5 · 0.7 ⋯ increase threshold 0.9 HIGH THRESHOLD Predict + only when extremely confident Only the strongest scores are treated as positive. Low TPR miss many real +’s Low FPR very few false alarms KEY INSIGHT You cannot increase TPR without also increasing FPR — unless the classifier improves.

2.8 Constructing the ROC Curve

The ROC (Receiver Operating Characteristic) curve was originally developed in the 1950s for signal detection theory — distinguishing a true radar signal from noise — before being adopted for classifier evaluation. It plots TPR (y-axis) against FPR (x-axis) as the decision threshold sweeps across its full range.

Step-by-step algorithm

  1. For every test instance, obtain the classifier's score s = P(+|X).
  2. Sort all instances in descending order of s (most confident + on top).
  3. Place a candidate threshold between every unique score value.
  4. At each candidate threshold, count TP, FP, TN, FN. Compute:
\( TPR = \text{Sensitivity (Recall)} = \frac{TP}{TP + FN} \qquad\text{(y-axis)} \)

\( FPR = 1 - \text{Specificity} = \frac{FP}{FP + TN} \qquad\text{(x-axis)} \)

Plot each (FPR, TPR) pair. Connect the dots. The result is the ROC curve. It always passes through (0,0) (threshold=1, predict nobody) and (1,1) (threshold=0, predict everybody). Best classifier = hugs the upper-left corner (0,1).

2.9 AUC — Area Under the ROC Curve

The ROC curve gives a picture of performance across all thresholds. To compare models numerically we summarise the whole curve by the area beneath it:

2.10 Comparing Models with ROC

AUC compares two models by a single number. When we instead need to compare two specific operating points, we compare their positions on the ROC plane directly.

Northwest dominance rule

Given two candidate thresholds (or two different models) plotted as points on the FPR-TPR plane:

Point A dominates point B iff A is strictly to the northwest of B: lower FPR AND higher TPR.

If neither dominates the other — one has higher TPR but also higher FPR — which is "better" depends on the per-error costs that the deployment environment assigns.

Example from lecture: Point (0.1, 0.6) dominates (0.2, 0.5) → lower FPR + higher TPR. But (0.1, 0.6) vs. (0.2, 0.7)? The latter has higher TPR but worse FPR, so deployment cost structure decides.

2.11 Choosing the Optimal Operating Threshold

Two automated methods to pick the single "best" point along the ROC curve.

Euclidean Distance Method
Youden's J Statistic

Distance from the perfect classifier at (FPR = 0, TPR = 1):

\( \text{Euc} = \sqrt{(1 - TPR)^2 + FPR^2} \)

We minimize Euc across all candidate ROC points, since it measures how close a point lies to the ideal corner. It is 0 for a perfect classifier and at most √2 ≈ 1.414 for a classifier sitting at (1, 0).

Euclidean distance from the perfect classifier corner A right triangle showing the distance from the ideal point (FPR=0, TPR=1) to an example operating point. FP Rate TP Rate (0, 1) perfect classifier (fprate, tprate) Euc

Euc is the straight-line distance between an operating point and the ideal top-left corner (0, 1).

The Youden index instead maximizes the vertical distance of the point above the diagonal. It is widely used in the medical literature.

\( J = TPR - FPR \quad\text{or equivalently}\quad Sensitivity + Specificity - 1 \)

Range [0, 1]. J = 1 only when perfect (TPR=1, FPR=0). J = 0 when the point sits on the random diagonal.

2.12 ROC Curve — What It's Used For (2 Main Jobs)

To summarise, the ROC curve serves two distinct purposes, and it is worth keeping them separate:

  1. Model comparison via AUC (area): Pick the classifier/model with the larger AUC. Good overall measure of ranking quality, threshold-agnostic.
  2. Threshold optimization: After choosing the model, find the point on that model's ROC curve that optimizes your deployment cost function (Euc, Youden, or custom cost-weighted FPR/TPR tradeoff) → set that threshold in production.

3. Interactive Examples

3.1 Imbalanced Dataset Metric Calculation

Classifier on medical diagnosis: 8 sick / 1000 patients total. Confusion matrix below:

Predicted Total
Sick (+) Healthy (−)
True Sick TP = 8 FN = 2 10
Healthy FP = 48 TN = 942 990
Total 56 944 1000
Accuracy = (TP + TN) / Total = (8 + 942) / 1000 = 95.0%

Recall = TP / (TP + FN) = 8 / (8 + 2) = 8/10 = 80.0%

Precision = TP / (TP + FP) = 8 / (8 + 48) = 8/56 ≈ 14.3%

F1 = 2 · (Precision · Recall) / (Precision + Recall) = 2 · (0.143 × 0.80) / (0.143 + 0.80) ≈ 2 × 0.114 / 0.943 ≈ 0.242 (24.2%)

Interpretation: 95% accuracy hides a terrible classifier for this specific task. An F1 of 0.24 reflects the disastrous precision (48 healthy people falsely told they are sick). In medical screening, high Recall (e.g., ≥ 95%) is usually the primary KPI to ensure we don't miss sick patients.

3.2 β Parameter Tuning

For each task, pick β ∈ {0.3, 1, 4} (low, equal, high) to weight Precision vs. Recall appropriately:

  1. Email spam filter: Blocking a real job-offer email (False Positive) is much worse than letting a spam email through (False Negative).
  2. Airport bomb-detection scanner: A missed bomb (False Negative) is catastrophic; a false positive just leads to a bag re-check.
  3. Generic document classification: Balanced classes, no obvious asymmetry between FP and FN costs.
  1. β = 0.3 (Low β emphasizes Precision). We must heavily penalize false positives (ham flagged as spam).
  2. β = 4 (High β emphasizes Recall). We must maximize true positives and tolerate moderate false alarms to ensure no bombs are missed.
  3. β = 1 (Standard F1). No cost asymmetry exists, so we weight Precision and Recall equally.

3.3 Threshold Sensitivity: Why 0.5 Isn't Special

Ten test instances with true class and classifier probability P(+|x):

Instance12345678910
P(+|x)0.950.930.870.850.800.780.760.530.430.25
True Class++−+−+−−−+

Using a 0.5 cutoff, every score at or above the threshold is predicted positive. But 0.5 is just one arbitrary choice among many. Compute the accuracy of this classifier at threshold = 0.5, then again at 0.87 and 0.43, and see whether 0.5 is really the best cutoff for this data.

At Threshold 0.5: Predict + for rows 1–8, − for 9, 10.
TP=4 (rows 1,2,4,6), FP=4 (rows 3,5,7,8), TN=1 (row 9), FN=1 (row 10).
Accuracy = (4+1)/10 = 50%

At Threshold 0.87: Predict + for rows 1,2,3.
TP=2, FP=1, TN=4, FN=3.
Accuracy = (2+4)/10 = 60%

At Threshold 0.43: Predict + for rows 1–9.
TP=4, FP=5, TN=0, FN=1.
Accuracy = (4+0)/10 = 40%

Conclusion: Threshold 0.87 happens to be better than 0.5 here, but checking three thresholds by hand doesn't tell us which cutoff is truly optimal. An ROC curve plots TPR vs. FPR at *all* thresholds so we can see every trade-off at once. Problem 3 in Section 4 (Numerical Solutions) builds that complete table from this same dataset, then finds the optimal threshold using Euc and Youden's J.

3.4 Northwest Point Comparison

Six candidate threshold points (FPR, TPR): (0.1, 0.6), (0.2, 0.5), (0.4, 0.2), (0.5, 0.5), (0.7, 0.7), (0.2, 0.7).

Six candidate ROC operating points A scatter plot of six FPR/TPR points against the random-classifier diagonal. FP Rate TP Rate 0 1 (0.1, 0.6) (0.2, 0.5) (0.4, 0.2) (0.5, 0.5) (0.7, 0.7) (0.2, 0.7)

Six operating points on the FPR–TPR plane, with the random-classifier diagonal for reference.

  1. Which one dominates (0.2, 0.5) using the northwest rule?
  2. Which pair(s) are incomparable?
  3. Compute Euclidean distance to (0,1) for (0.2, 0.7) and (0.1, 0.6). Which minimizes distance?

(a) (0.1, 0.6) dominates (0.2, 0.5) because it has a lower FPR (0.1 < 0.2) AND a higher TPR (0.6 > 0.5).

(b) (0.1, 0.6) and (0.2, 0.7) are incomparable. The second has higher TPR (+0.1) but worse FPR (+0.1). Neither is strictly better.

(c)

\( \text{Euc}(0.2, 0.7) = \sqrt{(0.2 - 0)^2 + (0.7 - 1)^2} = \sqrt{0.04 + 0.09} = \sqrt{0.13} \approx \mathbf{0.361} \)

\( \text{Euc}(0.1, 0.6) = \sqrt{(0.1 - 0)^2 + (0.6 - 1)^2} = \sqrt{0.01 + 0.16} = \sqrt{0.17} \approx \mathbf{0.412} \)

(0.2, 0.7) is closer to the northwest ideal (0,1) by Euclidean distance, so it would be selected over (0.1, 0.6) using this specific metric.

3.5 Python Demo: ROC Curve and Optimal Threshold

On a larger, more realistic dataset (a few hundred samples rather than 10 toy instances), the same construction produces a smooth ROC curve. Here we fit a classifier, sweep every threshold with sklearn.metrics.roc_curve, and use the Euclidean-distance method from §2.11 to pick the operating threshold automatically.

from sklearn.metrics import roc_curve, roc_auc_score import numpy as np # y_true: ground-truth labels, y_scores: P(+|x) from the classifier fpr, tpr, thresholds = roc_curve(y_true, y_scores) auc = roc_auc_score(y_true, y_scores) # Euclidean-distance method (Section 2.11) distances = np.sqrt((1 - tpr)**2 + fpr**2) optimal_idx = np.argmin(distances) optimal_threshold = thresholds[optimal_idx] print(f"AUC = {auc:.3f}") print(f"Optimal threshold (min distance) = {optimal_threshold:.3f}")
AUC = 0.880 Optimal threshold (min distance) = 0.263
ROC curve with optimal threshold from the Python demo A smooth ROC curve with area under the curve 0.880, with the distance-method optimal threshold of 0.263 marked. ROC Curve with Optimal Threshold False Positive Rate (FPR) True Positive Rate (TPR) Distance Method (threshold = 0.263) ROC Curve (AUC = 0.880) Random Classifier

The distance method picks the point closest to the ideal corner (0, 1) — here corresponding to a decision threshold of 0.263, not the default 0.5.

4. Numerical Solutions

Problem 1: Split Arithmetic from the Confusion Matrix

A binary classifier on 1,000 test samples produces: TP = 120, FN = 60, FP = 40, TN = 780.

  1. Build the 2×2 confusion matrix and verify the totals add up.
  2. Compute classification accuracy.
  3. Compute True Positive Rate (Recall/Sensitivity) and False Positive Rate.
📘 Show Solution

(a) Confusion matrix:

PredictedTotal
+−
True+120 (TP)60 (FN)180
−40 (FP)780 (TN)820
Total1608401000

(b) Accuracy = (120 + 780)/1000 = 0.900 (90%).

(c)

\( TPR = \frac{TP}{TP+FN} = \frac{120}{180} = 0.\overline{6} \approx 66.7\% \)
\( FPR = \frac{FP}{FP+TN} = \frac{40}{820} \approx 4.88\% \)

Problem 2: Full Confusion Matrix Derivation for Imbalanced Binary Classification

Classifier run on n = 500 samples, positive rate = 10% (50 sick / 450 healthy). Results: 40 sick correctly caught, 90 healthy incorrectly flagged.

  1. Fill in every cell of the confusion matrix (TP / FN / FP / TN).
  2. Compute Accuracy, Recall, Precision, F1.
  3. How would F2 (β = 2) differ from F1 here? Calculate F2 and compare directionally.
📘 Show Solution

(a) True positives: 40. Total real positives 50 → FN = 10. FP = 90 given. Total healthy = 450 → TN = 450 − 90 = 360. Matrix: TP 40 / FN 10 / FP 90 / TN 360.

(b)

\( \text{Acc} = \frac{40 + 360}{500} = \frac{400}{500} = 80.0\% \)
\( \text{Recall} = \frac{40}{40 + 10} = \frac{40}{50} = 80.0\% \)
\( \text{Precision} = \frac{40}{40 + 90} = \frac{40}{130} \approx 30.8\% \)
\( F_1 = 2 \cdot \frac{0.3077 \cdot 0.80}{0.3077 + 0.80} \approx \frac{0.4923}{1.1077} \approx 0.444 \)

(c)

\( F_2 = (1+4) \cdot \frac{P \cdot R}{4P + R} = 5 \cdot \frac{0.2462}{1.2308 + 0.80} \approx 5 \cdot \frac{0.2462}{2.0308} \approx 0.606 \)

F2 (≈ 0.606) is substantially higher than F1 (≈ 0.444) because β = 2 up-weights Recall, which this model does relatively well on (80%), while caring less about its poor Precision (30.8%). The "all caught but noisy" character of the model is rewarded as β grows.

Problem 3: Full ROC Table Construction

Continuing the same 10 test instances from §3.3 (Construct ROC by Hand), build the full ROC table by sweeping the threshold through every unique P(+|x) value. As a reminder, the dataset is:

Instance12345678910
P(+|x)0.950.930.870.850.800.780.760.530.430.25
True Class++−+−+−−−+

Each unique P(+|x) score becomes a threshold. For each one, count how many rows are predicted positive (score ≥ threshold) and derive TP/FP/TN/FN, then TPR, FPR, Euc, and Youden's J.

📘 Show Solution
Threshold ≥Predict +: rowsTPFPTNFNTPRFPREucJ (Youden)
1.00 (nobody)∅00550.00.01.0000.0
0.95{1}10540.20.00.8000.2
0.93{1,2}20530.40.00.6000.4
0.87{1,2,3}21430.40.20.6320.2
0.851–431420.60.20.4470.4
0.801–532320.60.40.5660.2
0.781–642310.80.40.4470.4
0.761–743210.80.60.6320.2
0.531–844110.80.80.8250.0
0.431–945010.81.01.020−0.2
0.25 (all)1–1055001.01.01.0000.0
ROC curve for the 10-instance worked example A step-function ROC curve built from the ten test instances, with the tied optimal points marked. ROC Curve — 10-Instance Example False Positive Rate (FPR) True Positive Rate (TPR) min Euc / max J (0.2, 0.6) min Euc / max J (0.4, 0.8) tied optimum

Step-function AUC for this n = 10 toy set ≈ 0.68 (trapezoidal rule). Compare to the smoother, larger-sample curve in the Python demo below (AUC = 0.880).

Minimum Euc = 0.447 occurs at two tied points: (0.85 threshold, FPR = 0.2, TPR = 0.6) and (0.78 threshold, FPR = 0.4, TPR = 0.8). A tie! The first has higher Precision (low FPR), the second has higher Recall (high TPR) — deployment cost structure picks between them.

Maximum Youden J = 0.4 is achieved by (0.93 threshold, 0.4/0), (0.85 threshold, 0.6/0.2), and (0.78 threshold, 0.8/0.4). A 3-way tie that reflects the small n = 10 test set.

Problem 4: Youden Index vs. Euclidean Distance

Two candidate threshold points on ROC: Point M = (FPR 0.15, TPR 0.75), Point N = (FPR 0, TPR 0.66).

  1. Compute Euc for M and N. Which minimizes it?
  2. Compute Youden J for M and N. Which maximizes it?
  3. Interpret: why do the two criteria disagree on which is "best"? When would each be preferred?
📘 Show Solution

(a)

Euc(M) = √((1−0.75)² + 0.15²) = √(0.0625 + 0.0225) = √0.085 ≈ 0.2915 ✅ smaller
Euc(N) = √((1−0.66)² + 0²) = √0.1156 ≈ 0.340

Euc prefers M.

(b)

J(M) = 0.75 − 0.15 = 0.60
J(N) = 0.66 − 0 = 0.66 ✅ larger

J prefers N — the two criteria disagree.

(c) Euc weights "distance from the corner" in squared Euclidean space, so small movements near the TPR = 1 axis count more. Youden treats TPR and FPR linearly equal. Here, N buys a lower FPR (0.15 → 0) at the cost of some TPR (0.75 → 0.66); J calls this a good trade since it just adds the two changes, but Euc calls it a bad trade since squaring makes the TPR loss weigh more than the FPR gain. If missing a positive (low TPR) and a false alarm (high FPR) cost literally the same dollar amount, use J. If getting near-perfect TPR is disproportionately important (medical), Euc (or the equivalent β-heavy Fβ metric) is more natural.

5. Try It Yourself

Problem 1: Imbalance Metrics Practice

Ad-tech task: Out of 10,000 ad impressions, only 100 users click (positive). Our model predicts 150 clicks total. Of its 150 predicted clicks, 60 are real (TP) and 90 are wrong (FP). Of the 100 real clicks it missed 40 (FN).

  1. Fill in TP, FN, FP, TN.
  2. Calculate accuracy, precision, recall, F1, F0.5 (β = 0.5 — penalize FP more).
  3. Interpret: Why is F0.5 lower than F1 here?
📘 Show Solution

(a) TP = 60; FN = 40; FP = 90; TN = 10000 − 60 − 40 − 90 = 9810.

(b)

\( \text{Acc} = \frac{60 + 9810}{10000} = 98.7\% \)
\( P = 60/(60+90) = 40.0\%; \;\; R = 60/(60+40) = 60.0\% \)
\( F_1 = 2 \cdot \frac{0.40 \cdot 0.60}{0.40 + 0.60} = 2 \cdot 0.24 = \mathbf{0.480} \)
\( F_{0.5} = 1.25 \cdot \frac{0.24}{0.25 \cdot 0.40 + 0.60} = 1.25 \cdot \frac{0.24}{0.70} \approx \mathbf{0.429} \)

(c) F0.5 ≈ 0.429 < F1 ≈ 0.480 because β < 1 weights precision more heavily. This model has P = 40% (worse) and R = 60% (better) — downgrading the good metric and upgrading the bad one makes the harmonic mean drop, which correctly reflects the advertiser's pain of wasting budget on 90 non-clickers for every 60 real clicks.

Problem 2: AUC Rank Interpretation

A model reports AUC = 0.85 on a binary classification test set with 500 positives and 500 negatives. Suppose I take a uniformly random positive and a uniformly random negative and compare their P(+|X) scores. What's the probability the positive's score is strictly greater? If I compare 100 independent positive-negative pairs, how many do I expect to be correctly ordered?

📘 Show Solution

Probability of correct ordering = AUC = 85% (that is exactly the probabilistic interpretation of AUC!). Expected number out of 100 independent pairs = 100 × 0.85 = 85 correctly ordered.

Problem 3: Optimal Threshold Selection

Four ROC operating points with (FPR, TPR) = (0.02, 0.60), (0.05, 0.80), (0.20, 0.96), (0.50, 0.99). Find: (a) Euclidean-minimizing, (b) Youden J-maximizing, (c) the choice for a costly-miss disease screening where TPR is 5× more important than FPR, and (d) the choice for a spam filter where FP (good→spam) costs 10× a FN (spam in inbox).

📘 Show Solution

Eucs: (0.02, 0.60) → √(0.4² + 0.02²) = 0.4005; (0.05, 0.80) → √(0.2² + 0.05²) = 0.206; (0.20, 0.96) → √(0.04² + 0.20²) ≈ 0.204; (0.50, 0.99) → √(0.01 + 0.25) = 0.51. (a) Min Euc ≈ (0.20, 0.96) (by a hair over 0.80/0.05).

Js: 0.58, 0.75, 0.76, 0.49. (b) Max J = (0.20, 0.96).

(c) Disease screening: TPR dominates. Choose (0.20, 0.96) → 96% of cases caught, accepting 20% false alarm rate (which is manageable, it just means more tests). If you can go even higher TPR at any cost, pick (0.50, 0.99).

(d) Spam filter: FPR cost dominates. Pick the lowest achievable FPR point that still catches meaningful spam: (0.02, 0.60). Only 2% of ham goes to spam folder. You miss 40% of spam (that's the tradeoff) — add a second layer or accept it.

6. Interactive Quiz

Answer all 6 questions. Click an option for instant feedback.

Your score: 0 / 6

7. Key Takeaways

  1. Accuracy lies on imbalanced data. Always report Precision, Recall, and F1/Fβ alongside accuracy when the minority class rate is far below 50%.
  2. β is the Precision/Recall knob. Low β prioritizes Precision (spam filter), high β prioritizes Recall (fraud, disease, bomb detection), and β = 1 gives standard F1 with equal weighting.
  3. Harmonic mean, not arithmetic mean. It pulls toward the smaller of Precision and Recall, so a model can't hide a weak score behind a strong one.
  4. ROC and AUC are threshold-free. The ROC curve plots TPR against FPR across every possible threshold, and AUC summarizes that curve as a single measure of ranking quality.
  5. Choosing a threshold is a separate step from choosing a model. Euc and Youden's J are two ways to pick the deployment threshold once a model is fixed, and they can disagree depending on how false positives and false negatives are costed.

8. Common Pitfalls

  1. Trusting accuracy only on imbalanced tasks. A "99% accurate" fraud detector can catch zero frauds. Always check the confusion matrix and Precision/Recall/F1 as well.
  2. Averaging Precision and Recall arithmetically. This gives 0.5 even when one is 100% and the other is 0%. The harmonic mean (F1/Fβ) collapses to 0 in that case, which is the correct verdict.
  3. Defaulting to F1 when the costs aren't equal. Use Fβ with a task-appropriate β instead. F1 assumes false positives and false negatives cost the same, which is rarely true in practice.
  4. Leaving the threshold at 0.5 without checking. The entire point of the ROC curve is that 0.5 is rarely the optimal cutoff for a given deployment cost structure.